Diversity of reproduction time scale promotes cooperation in spatial prisoner's 

dilemma games 
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We study an evolutionary spatial prisoner's dilemma game where the fitness of the players is 
determined by both the payoflts from the current interaction and their history. We consider the 
situation where the selection timescale is slower than the interaction timescale. This is done by 
implementing probabilistic reproduction on an individual level. We observe that both too fast and 
too slow reproduction rates hamper the emergence of cooperation. In other words, there exists an 
intermediate selection timescale that maximizes cooperation. Another factor we find to promote 
cooperation is a diversity of reproduction timescales. 



I. INTRODUCTION 



Many biological, social, and economic systems are built 
and sustained by cooperating individuals [l], Q . The is- 
sue of how cooperation emerges and persists among self- 
ish individuals is the topic of game theory [Hi]. The 
simplest and most versatile framework for evolutionary 
game theory is perhaps the so called symmetric 2x2 
games [1, 0, [1]. In such games, agents can employ two 
behaviors or strategies — cooperative or defective — in 
their interaction with others to optimize their fitness. 
The goal of this branch of game theory is to establish the 
conditions for cooperation (or, equivalently, defection) 
to spread. An important 2x2 game is the prisoner's 
dilemma (PD) that describes the situation when cooper- 
ation gives the highest population- level payoff but, in a 
short timescale, defection maximizes the expected payoff 
of an individual. In well-mixed populations, cooperators 
cannot outperform defectors and are doomed to extinc- 
tion 0. There are, however, many arguably realistic 
factors that favor cooperation. For example, one such 
factor is space — if the players are embedded in space, 
allowed only to interact with their neighbors, cooperation 
can spread in the population [ol, [ill [11, 111, [ll ■ The 
microscopic explanation for this phenomenon is that co- 
operators can form spatial clusters where the boundaries, 
although exploited by defectors, protect the cooperators 
within [13, [HI. On complex networks — scale- free 
networks in particular — cooperators have a tendency 
to occu py h igh degree nodes, which stabilizes coopera- 
tion [Ij, [l4l. For detailed surveys of this field, we 
refer to Refs. [isl. [l6l |. From these examples we learn 
that the spatial organization, the interaction patterns, 
are important elements behind the emergence of cooper- 
ation. This raises the question: are there also temporal 
patterns promoting cooperation? In a recent work 17], 
Chen and Wang investigated the effects of what they call 



"payoff aspiration" (roughly, the lower limit of the payoff 
before a player gets impatient changes strategy). They 
found that intermediate aspiration levels maximize co- 
operation and argue that too high levels cause too fast 
strategy changes, while too low levels make the agents 
not adaptive enough. These results suggest a more fun- 
damental principle — to achieve cooperation, there might 
exist an optimal timescale for updating strategies. In the 
present paper, we address this hypothesis by studying 
evolutionary spatial PD games and by explicitly consid- 
ering the time scales of interaction and selection. 



One of our basic assumptions is that the selection 
timescale is slower than the interaction timescale. This 
means that even though each player has a finite lifetime, 
it can interact many times before selection occurs. Dur- 
ing their lifetimes, agents do not change their strategies 
(or phenotype, in a more biological parlance). 



How can we connect the payoff of the game to fitness? 
The meaning of payoff is more problem-dependent than 
fitness — payoff is some result of the interaction that is 
correlated with reproduction ability [111 llSlI and often 
equated, mathematically, with fitness (a. I 111 [T5|. Some 
works using non-standard relationships between payoff 
and fitness are Ref. jl^ (where a background noise is 
added to the payoff to obtain fitness) and Ref. (study- 
ing non- linear fitness dependence on payoff). Though 
there are many possible ways to characterize the rela- 
tionship between payoff and fitness, in our model we as- 
sume the two quantities are positively correlated (like in 
natural selection), and also including an inheritable fit- 
ness component (so called maternal effect [2l|), which can 
appear due to social, environmental or epigenetic mech- 
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In the rest of the paper, we first introduce our model, 
defining the fitness and reproduction time scale. Then 
we present our ntimerical and analytic results in detail 
and relate these to other studies. 
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II. THE MODEL 

A. Payoff and fitness 

In the present study we consider evolutionary games 
where individuals, located on a two-dimensional square 
lattice, play PD games with their closest neighbors. Fol- 
lowing common practice , we consider pure strate- 
gies without self-interaction (where one player also plays 
with oneself [9]), and use the following parameterization 
of the PD: if two players both cooperate (C) then both 
get a "reward" of 1, two defectors (-D) both obtain the 
"punishment" , a C meeting a D receives "the sucker's 
payoff" 0, while D acquires "the temptation to defect", 
h G [1,2]. For each site i we define its fitness at genera- 
tion t as fi{t) = afi{t — 1) + gi — a gi, where i > is the 
generation oi i. gi is the instantaneous payoff from the 
game and the parameter a G (0, 1), the heritability, sets 
the balance between the present and past payoff gains 
— the relative importance of a previous generations, or 
strength of maternal effects [2l| decays with a factor a 
per time step. 



B. Reproduction 

Since our main focus is to evaluate how the diversity 
of reproduction timescale affects the evolution of coop- 
eration, we discard the normal setup that strategy up- 
dating of the players is implemented immediately after a 
round of game [l5j . Instead, we follow the approach of 
Ref. 22], and assume that the timescale of selection is 
slower than that of the interaction. In particular, we up- 
date the population by using probabilistic dynamics: for 
each focal site i, a reproduction event occurs with prob- 
ability p, whose magnitude therefore characterizes the 
timescale of selection. When p is close to unity, selection 
is frequent, while for small values of p the reproduction 
event is rare and natural selection is slow. It is easy to see 
that the average time of selection on each site is 1/p. By 
this simple way, we are also able to introduce diversity 
of reproduction timescales. 

When strategies are updated, the focal player i and one 
randomly selected neighbor, say j, compete for creating 
an offspring at i. The focal site will, depending on their 
fitnesses, be occupied by either the offspring of i or j. To 
be specific, the neighbor j places a new offspring with 
the same strategy on the focal site i with a probability 

l + exp[(/, 

where k measures the probabilistic selection intensity (or 
noise, or temperature); otherwise i is replaced with an 
offspring of the previous individual at z. A newborn is 
assigned the initial fitness fi{0) — 1 (the precise value 
of fi{0) does not affect our conclusions). An alternative 
interpretation is that the individuals sometimes can start 



with new strategies taken from a neighbor with higher 
fitness than the individual. 



C. Effects of noise 

The noise parameter k is known to have a nontriv- 
ial effect on the evolutionary trajectories [l^, [13, |25| . 
More specifically, the highest cooperation levels appear 
in moderate amounts of noise. (Here we want to point 
out such optimal behavior of noise in promoting coop- 
eration is not universal, but dependent on the topology 
of the connectivity structure.) Furthermore, in a very 
recent paper (26j , Szabo et al. reported an interesting re- 
sult that the Darwinian selection of "temperature" favors 
values of k close to 0.1 which is related to the optimum 
providing the highest level of cooperation. In this paper 
we will not investigate the effects of k further, but set k 
to a value, 0.01, that is low enough for the effects of the 
game to dominate the effects of noise, and high enough 
for selection to occur. 



D. Parameter values and simulation technicalities 

We investigate our model with Monte Carlo (MC) sim- 
ulations. All runs are on a 100 x 100 square lattice with 
periodic boundary conditions. Initially, either a coopera- 
tor or a defector, randomly chosen with equal probability, 
occupies each site. During one MC sweep we go through 
all players once in a random order. The key observable is 
the average fraction Fc of cooperators in the population 
in the stationary state, which is measured for the last 
10000 sweeps of the total simulation time 6 x lO'*. The 
results presented below are averages over twenty realiza- 
tions from independent initial configurations. 

III. RESULTS AND ANALYSIS 
A. Monomorphic populations 

We start by investigating how the heritability a affects 
the evolution of cooperation. The simulation results for 
Fc as a function of b for three values of a and two values 
oi p are shown in Fig. [TJ In general, we can observe that 
with an increasing a, the average fraction of cooperation 
grows. For sufficiently large a, the system can even, for 
the weakest temptations to defect, reach the absorbing 
state of full cooperation. This means that, the ability 
of inheritance of fitness from previous generations has a 
positive impact on the emergence of cooperation. The 
larger a is, the greater is the cooperation level. This re- 
sult is somewhat expected since other studies have shown 
that the average payoff of C in the equilibrium is greater 
than that of D [22j . 

Even if the a-dependence of Fc is qualitatively ex- 
pected, it is rather different for the two p- values of 
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FIG. 1: (Color online) Average fraction Fc of cooperation as 
a function of b for different values of a. In (a) and (b) the 
individuals have reproduction rate p = 0.1 and 0.5, respec- 
tively. 
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FIG. 2: (Color online) Average fraction Fc of cooperators as 
a function of b for different values of p. The parameter a is 
fixed to 0.9. 



Fig. [Ha) and (b). In Fig. [2] we fix a = 0.9 and study how 
the average fraction of cooperation evolves as p changes. 
For large [p = 0.9) and small {p = 0.05) values Fc is 
smaller than for p = 0.1. This result suggests that a 
moderate reproduction rate promotes the spread of co- 
operation. To make this point clearer, in Fig. [3] we sum- 
marize the results of Fc as a function of p for different 
values of a. As surmised, the highest cooperation level is 
achieved in a region where p is moderate, resembling co- 
herence resonance phenomenon — that p alone (without 
e.g. any weak periodic force) can promote cooperation, as 
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FIG. 3: (Color online) Average fraction of cooperation Fc as 
a function of reproduction rate p for b = 1.02. The squares, 
circles, and triangles correspond to a = 0.1, 0.5, and 0.9, 
respectively. 

has been found frequently in recent literatures 0, [2^ . 
For example, by introducing additive noise to the pay- 
off matrix of the PD game, Fere found that coopera- 
tion is maximized at a moderate variance of payoff vari- 
ations |2J]. Along a different research line, Ren et al. 
studied PD game on random regular graphs and found 
that there exists an optimal amount of topological ran- 
domness, leading to the highest level of cooperation [2^ . 
Going back to our study, note that the optimal value 
Po, where Fc is maximized, depends on the value a. In- 
creasing a will shift po toward smaller values (smaller 
reproduction rates). For instance, for a = 0.1 we have 
Po ~ 0.41, while for a = 0.9 Po is about 0.12. This point 
is also reflected by Figs. [Hand [21 where given a = 0.9 the 
curves for p — 0.1 take the rightmost position. 

The nontrivial behavior of Fc in Figs. [TJ [21 and [3| can 
be qualitatively understood as follows. As p — > 1 — 
selection happens frequently; the average fitness of C 
should be lower than that of D (due to the frequent 
exploitation on C by D), so Fc will decrease. On the 
other hand, in case of p ^ — selection is very slow; 
defectors around the boundary have enough time to ob- 
tain a fitness high enough to outcompete the cooperators. 
This is reminiscent of the observation that the formation 
of C-clusters can protect cooperation in the spatial PD, 
whereas the clusters of D would deteriorate their own 
environment [l^, [H, H^]. In the light of this, the maxi- 
mal cooperation levels are reached for p-values that are 
neither be too large nor too small, so that C has a larger 
chance than D to obtain high fitness. In order to investi- 
gate this hypothesis further, we will first check what will 
happen if the population is well mixed rather than two- 
dimensionally extended. Though mean-field treatment 
is not appropriate for spatial case, it is a fair first-order 
approximation of the well-mixed case. 

For simplicity we assume that the average coordination 
number (number of interactions per site, per MC sweep) 
is z. In the mean-field approximation we can write the 
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following equation for the evolution of the average frac- tion of cooperation: 



dF 

= {1 - F,)F,Wd^c ~ F,{1 ~ F,)Wc-,D ^ F,{1 



-Fc) 



-FJl-F, 



) tanh 



f D fc 
2k 



1 + exp[(/^ - fc)/K] 1 + exp[(/c - fo)/^] 



(2) 



where fcjj denote the average fitness of C and D. The 

sign of fjj — Jq determines whether Fc converges to 1, 0, 
or remains unchanged. Since each individual reproduces 
with probability p, the average reproduction period is 
T = where [■] represent the smallest integer larger 

than the argument. The average fitness of D at age t 
can be written as fu{t) = af jj{t — 1) + zhFc- Assume 
the age of the individuals are equally distributed in the 
region t G [^,T], using the boundary condition f ^{0) = 
1, we obtain the average expected fitness of D: fjj = 
J2i=o f D (^) ■ After some algebra, we get 



T-l 



T-l 



T-l T-l i 



1=1 



i=0 j=Q 



(3a) 
(3b) 

We can see that for any b > 1, because fj^ — fc > 0, 
Fc will go to zero. The incorporation of spatial ef- 
fects or diversity of reproduction times will change this 
picture. The permission of only nearest-neighbor in- 
teraction, and non-synchronous reproduction, also as- 
sures the possibility of fjj — < at a local spatial 
scale. Another factor is that D clusters "defeat them- 
selves" [13, m, [131 . combination with these obser- 
vations, the intrinsic stochasticity of probabilistic repro- 
duction gives C a chance to outcompete D for dispersion, 
given that p is appropriately selected. As a consequence, 
cooperators are favored by diversity of reproduction, and 
can even dominate the whole population provided that b 
is sufficiently small. 



B. Polymorphic populations 

In reality there is a spread in reproduction times. We 
proceed to investigate the effects of diversity on our PD 
model. For simplicity, we restrict ourselves to a bimodal 
distribution of p- values: we assign p = 0.1 to a fraction 
V of the players and p — 0.9 to the rest. 

In Fig.|4]we show the dependence of Fc on b for a = 0.9 
and different values of v. We find that for appropriate 
mixing of the two types of individuals, for instance v = 
0.5 and 0.7, the final cooperative level is further enhanced 
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FIG. 4: Average fraction of cooperation Fc as a function of b 
for different systems where fraction v of population has repro- 
duction rate 0.1, and the others have p = 0.9. The parameter 
a is set to 0.9. 



as compared to a monomorphic population (u = or 1). 
To investigate this further, in Fig. [5] we graph Fc{b) for 
three systems: one with a polymorphic population where 
half of the population has p = 0.1 the other has p = 
0.9, another with a monomorphic population of the same 
average reproduction rate p — 0.5 as the polymorphic 
population, and a third monomorphic population with 
the same average life time as the polymorphic population 
(giving p = 0.18). In this comparison, the polymorphic 
population has a higher cooperation level than both the 
monomorphic reference populations. 

From Fig. 31 we also note that an intermediate mixing 
of reproduction rate of the population benefits the main- 
tenance of cooperation. This finding is corroborated by 
Fig. [6] where Fc{v) is displayed for various 6- values. For 
small b, Fc can reach unity even for moderate values of 
V. With an increasing temptation to defect, the region of 
V with non-zero cooperation vanishes. Despite this, the 
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FIG. 5: (Color online) Average fraction Fc of cooperators as 
a function of b for three special cases where all players have 
a reproduction rate p = 0.5 (squares), p = 0.18 (circles), and 
50% individuals with p = 0.1 and the remaining with p = 0.9 
(triangles). Note that the population with half p = 0.1 and 
half p = 0.9 have the same average reproduction rate as the 
population with p = 0.5, and the same average life span as 
the population with p = 0.18. 
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FIG. 6: (Color online) Average fraction of cooperators as a 
function of the balance between reproduction rates v for dif- 
ferent values of 6. u = or 1 represent monomorphic popu- 
lations; diversity (measured in standard deviation) is largest 
for V = 1/2. a is set to parameter 0.9. 
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FIG. 7: (Color online) The cumulative distribution of the 
fitness of agents with different p- values in a mix of half p = 0.1, 
half p — 0.9. (a) shows the fitness for agents with p — 0.1, 
(b) shows curves for agents with p = 0.9. Solid and dashed 
lines are for C, D, respectively. Other parameters: a = 0.9, 
b = 1.1. Under such parameterization, the final fraction of C 
with p = 0.1, and p = 0.9, is about 0.38(2), and 0.33(2); the 
corresponding D with p = 0.1, and p = 0.9, is about 0.12(2), 
and 0.17(2), respectively. 



tion rate. This point can be verified by the data presented 
in Fig. [71 where the cumulative distribution of the fitness 
of the population is shown. In this sense, even though the 
microscopic mechanisms are different, both our present 
model and that of Ref. [2^ can be regarded as belong- 
ing to a generalized class that adjusts the temporal or- 
ganization of the individuals in a way that maximizes 
cooperation. 



peak of Fc is located at intermediate p-values. 

The enhancement of cooperation at intermediate lev- 
els of mixing of two types of individuals can be explained 
in the same vein as Ref. \2§\, where Szolnoki and Szabo 
studied evolutionary PD game with a fraction v of play- 
ers having higher influence to spread their strategies than 
the others. They found that highest level of cooperation 
is attained when v is around 0.5. Their interpretation 
was that the less influential individuals act as follow- 
ers [2^ which stabilizes cooperation. In our case, by 
analogy, we can assume individuals with high reproduc- 
tion (large p) act as followers [28[, since, on average, they 
always get lower fitness than those with slow reproduc- 



To further investigate how the difference of reproduc- 
tion rate affects the evolution of cooperation; we mea- 
sure the w-dependence of extinction threshold of bo and 
be — the values of b that demarcate the region of coex- 
isting cooperators and defectors. The simulation results 
are shown in Fig. [51 Between the upper and lower critical 
points C and D can coexist. Above (below) this region 
only defectors (cooperators) remain alive in the final sta- 
tionary states. Note that, for the sake of comparison, we 
extend the 6-values to the region smaller than 1 where 
the game is a Stag- Hunt game rather than a PD p^ . 
From this figure, we observe that for intermediate v both 
bjj and be attain their maximum values, though not at 
the same v. 
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FIG. 8: (Color online) The extinction thresholds he, bo of 
cooperators and defectors, respectively, as a function of v. 
The parameter a — 0.9. Note that the range of b extends to 
values lower than 1, where the game is not a proper prisoner's 
dilemma. 



IV. CONCLUSIONS AND DISCUSSION 

To summarize, in the present work we have proposed 
and analyzed an evolutionary, spatial prisoner's dilemma 
game including a partial inheritance of fitness, a prob- 
abilistic reproduction, and a diversity of reproduction 
rates. The combination of these factors gives rise to 
rich dynamic behavior of the system. In particular, the 
stronger the ability of inheritance of fitness is, the more 
cooperation is there. The reproduction rate has a non- 
trivial effect on the evolution of cooperation. When the 
parameter a (characterizing the strength of inheritance 
of fitness) is fixed, the highest cooperation level is re- 
alized for moderate reproduction rates. Both too slow 
and too fast reproduction rates can harm cooperation. 
Moreover, we find that the presence of diversity of re- 
production rate can enhance cooperation. The behavior 
of promoting cooperation is found to resemble coherence 
resonance. These findings suggest that the heterogeneity 
of individual traits might benefit cooperation in PD situ- 
ation, giving another clue to the emergence of population 
in social and biological systems of selfish individuals. 

In a recent work, Qin et al. i29|] explored the effects 



of infinite memory in a spatial PD game (allowing self- 
interaction). They found that with a moderate mem- 
ory effect, the extinction threshold ho reaches a maximal 
value, but observed no such behavior for he- Their mem- 
ory mechanism is similar to the maternal-effect fitness 
inheritance in our model. In another related work [23 |. 
Roca et al. considered independent interaction and se- 
lection timescales in evolutionary games. The authors 
studied all 12 nonequivalent symmetric 2x2 games, and 
found that, depending on the type of the game, rapid se- 
lection may lead to changes of the asymptotic equilibria 
and changes in the basins of attraction Ref. [l^l 

differs from ours in the approach to strategy updates — 
they model updates as a Moran process, whereas we use 
a stochastic migration rule; and in the interaction con- 
figuration they, in contrast to us, consider a non-spatial 
(or well-mixed) population. 

One future problem is how cooperators vanish as the 
temptation to defect increases. For evolutionary PD with 
stochastic imitation rules Szabo and Toke [lO| found a 
phase transition of the directed percolation universality 
class [13. We expect a similar transition can occur in 
our model in the case of large value of p. However, this 
point needs to be confirmed in further studies. Moreover, 
how the p-value maximizing Fc depends on the param- 
eter a is not addressed in the present work. The work 
in [12] suggests that large degree nodes are important 
for promoting cooperation. In a recent paper [30|, mid- 
dle degree nodes, on the other hand, are found to be 
crucial for stabilizing cooperation. Along these lines, we 
expect that the integration of inhomogencous interaction 
pattern and reproductive diversity to our present model 
would be interesting. 
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